Punchlines

• From priors survey, many expected PPS / HT to do well and other approaches to underestimate
• In fact PPS / HT seems to dominate both in terms of MSE and cost
• Biases appear not to be homogeneous across studies, counter to our model
• Model that allows heterogeneity suggests that heterogeneity in relative bias of methods is large
• Models provide adjusted estimates for each team: these are mostly similar across models, though assumption dependent
  
• We still have relatively weak overlap in methods across studies: this means still some sensitivity to priors especially for heterogeneity model

PRIF meta-analysis

PRIF meta-analysis

• Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

• Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs

PRIF meta-analysis

• Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

• Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs

• Approach: Use MIDA to provide both meta-analysis of PRIF studies and simulations for study design diagnosis

PRIF meta-analysis

• Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

• Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs

• Approach: Use MIDA to provide both meta-analysis of PRIF studies and simulations for study design diagnosis

• Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta

Disagreements about bias

Distribution of expected bias across respondents

• Most respondents expected that most methods would underestimate prevalence
• Some expected rds would especially underestimate prevalence
• One expected all methods would be unbiased
• PPS-HT has the tightest distribution centered close to 1

Uncertainty about bias

Average bias expectation with average lower and upper bounds of credibility intervals

• There is considerable ex ante uncertainty regarding the performance of methods
• In almost all cases “average bounds” include 1
• Only for TLS-NSUM was the “average” credibility interval entirely above 1

Data

• Sample of estimates depends on estimands: prevalence (threshold 1 / threshold 2 / both), size (hidden / known)
  • e.g. for prevalence (threshold 1)

Main meta-analysis results: Threshold 1 (Biases)

prif_estimates |> 
  filter(inquiry == "hidden_prev1") |> 
  get_meta_estimates()

Main meta-analysis results: Threshold 1 (Prevalence)

Main meta-analysis results: Threshold 2 (Biases)

prif_estimates |> 
  filter(inquiry == "hidden_prev2") |> 
  get_meta_estimates()

Main meta-analysis results: Threshold 2 (Prevalence)

Estimated bias vs. priors

Estimated bias vs. bias priors

• Estimated biases exhibit more variance than priors on biases

Approach

Instead of assuming that the relative bias of a method \(\beta_{j}\) is constant, we assume that at each site it is a draw from a distribution: \[\beta_{j} \sim f(\beta^*_{j}, \sigma_{j})\] We get to estimate both \(\beta^*_{j}\) – the expected relative bias, and \(\sigma_{j}\): the heterogeneity in relative bias.

Bias comparisons across models

Estimated bias comparisons

• These correlate. Basic message though is that the heterogeneous model is consistent with strong heterogeneity: We entertain the possibility that every method could be an overestimate or an underestimate.

• This model has many parameters for not so many data points \(\Rightarrow\) prior sensitivity

Prevalence estimates from two models

Prevalence estimates from two models

• Biggest difference is in Brazil which sees less adjustment in heterogeneous model

Cost-benefit analysis by study

• Note that the frontier is covered largely by PPS. LTS/RDS+ (Tanzania) is also on the frontier here with low cost, but large error (from our estimates).

Cost-benefit analysis by strategy

Interactive meta-analysis

• The meta-analysis results are preliminary:
  • Estimates from some teams were not included in the meta-analysis dataset
  • The costs of each method might need to be adjusted
  • The priors on study level inquiries as well as relative biases might need to be adjusted
• We built an interactive Shiny dashboard for everyone to be able to explore and adjust data and priors
• We encourage everyone to access the dashboard and play with it here: shiny2.wzb.eu/syunyaev/hiddenmetashiny/

Interactive meta-analysis

Worries and todos

Todos

• Replication work not completed: please bring your data!
• Still relatively low overlap (\(n\) per method); we want to improve this (bring your data and lets implement the NSUMs)
• Implement structured heterogeneity analysis: for instance, based on types of networks

Worries and issues

• We are seeing some variation in results as a function of thresholds
• Seems clear we need to deviate from PAP
• We are assuming independence of samples in each study but this is not right

Onwards

• We are setting up the dashboard so that you can add more data and the meta-analysis updates automatically.

• The dream is as you do more studies you use multiple methods and you feed into this “rolling meta-analysis”

  • Who will curate?
  • What documentation is needed to join the club?

Conclusion and reflections

High level conclusion

• We know different people here love RDS, Link tracing, and TLS
• There are good prior reasons for each of these
• But meta-analysis suggests HT (more than) competitive
  • Others seem to underestimate in general
  • Others appear costly

Model performance

• Bias appear to be demonstrably not homogeneous across studies, counter to our pre-registered model
• Sensitivity to priors, thresholds

Reflections

• Even if estimates similar, if these are getting at different parts of the population then these are not substitutable: approach to get prevalence of subgroups within target population
• What finer data to use to assess optimal “mixing”?
• What else to take into account for decision between methods (e.g. Pakistan team described RDS as safer–this not captured in our analysis. )